Papers with selective fine-tuning
Divergence-Based Domain Transferability for Zero-Shot Classification (2023.findings-eacl)
Copied to clipboard
| Challenge: | a recent study shows that fine-tuning of neural models can improve performance on language-based tasks without brute-force searching effective task combinations. |
| Approach: | They propose to use divergence measures to estimate whether one task pair will perform better than another . they use 58 tasks and 6,600 task pair combinations to study the effect of different tuning methods . |
| Outcome: | The proposed method reduces end-to-end runtime by 40% by estimating transferability . the proposed method is based on 58 tasks and over 6,600 task pair combinations . |
Where CoT Reasoning Commits: Entropy Traces Identify Interpretable Attention Heads (2026.findings-acl)
Copied to clipboard
| Challenge: | a growing body of work suggests a disconnect between the generated rationale and the model's actual choice. |
| Approach: | They propose a mechanism-aware framework that interprets the evolving "choice state" of attention heads during CoT generation . they identify a set of intervention targets and perform Selective Head Fine-Tuning . |
| Outcome: | The proposed framework interprets the "choice state" of attention heads during CoT generation . it detects two functional behaviors: Steadfast Heads and Wavering Heads . |
On Surgical Fine-tuning for Language Encoders (2023.findings-emnlp)
Copied to clipboard
Abhilasha Lodha, Gayatri Belapurkar, Saloni Chalkapurkar, Yuanming Tao, Reshmi Ghosh, Samyadeep Basu, Dmitrii Petrov, Soundararajan Srinivasan
| Challenge: | preserving knowledge of target distribution by fine-tuning all layers can be expensive and may increase data volume requirements. |
| Approach: | They propose an efficient metric based on the diagonal of the Fisher information matrix (FIM score) to select the candidate layers for selective fine-tuning. |
| Outcome: | The proposed metric can select layers leading to strong performance on GLUE and SuperGLUE tasks and across distinct language encoders. |
How Do LLMs and VLMs Understand Viewpoint Rotation Without Vision? An Interpretability Study (2026.acl-long)
Copied to clipboard
Zhen Yang, Ping Jian, Zhongbin Guo, Zuming Zhang, Chengzhi Li, Yonghong Deng, Xinyue Zhang, Wenpeng Lu
| Challenge: | Existing studies on spatial intelligence from the perspective of visual-spatial intelligence have not explored whether visual intelligence alone is sufficient to endow models with spatial intelligence. |
| Approach: | They propose to use a linguistic perspective to investigate spatial intelligence from a theoretical perspective. |
| Outcome: | The proposed model performs poorly on the proposed dataset while human can easily achieve 100% accuracy. |
Tracing and Dissecting How LLMs Recall Factual Knowledge for Real World Questions (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have shown promising ability to perform commonsense reasoning. |
| Approach: | They propose a two-dimensional analysis framework that incorporates token back-tracing and token decoding to uncover how LLMs conduct factual knowledge recall. |
| Outcome: | The proposed framework shows that LLMs lack relevant knowledge but struggle to select the most accurate information based on context during the retrieval and rerank phase. |